@Neer393 I don't understand what have you optimized here.
You are still doing multiple calls: 1 to get table names and another to get table objects. Why not get table objects directly?

Also, have you considered the memory impact when loading everything into the heap? You could have iterated over TableIterable instead. I don't think that is a robust solution, it can potentially lead to OOM.
cc @dengzhhu653, @wecharyu

The earlier implementation had one msc call for getting table names and then one msc call each for getting the HMS table object for each table name.

The newer implementation reduces the msc calls in a way that one msc call is made for getting all table names and then using TableIterable, the number of msc calls for getting table objects becomes Number of tables / (BATCH_MAX_RETRIEVE config value [Default is 300])

So in the older implementation number of msc calls = 1 + number of tables
whereas in the newer implementation number of msc calls = 1 + (number of tables / [BATCH_MAX_RETRIEVE])

In my earlier implementation I had the same proposal of directly getting table objects where I had implemented direct HMS API endpoint like listTableNamesByFilter but the idea was dropped by @vikramahuja1001

in order to use batching, you need to have the table list to fetch - that's ok. However, instead of working with the batches, you load everything into memory.
Could you refactor to use Iterable (i.e make getTables return Iterable<Table>)?

Okay so for this fix should I create a new JIRA or as I am working on https://issues.apache.org/jira/browse/HIVE-28974 which is related to IcebergHouseKeeperService only should I attach the fix in this JIRA ?
Whatever you suggest is fine to me

+1 Use the TableIterator is more reasonable to avoid possible OOM.

Let me summarize the points.

• This PR would reduce # of API calls from O(num-tables) to O(num-tables / batch size). It's neat 👍
• This PR would require the O(num-tables * tbl-size) space. We'd like to reduce it
• Regardless of this PR, we retain the O(num-tables * length-of-table-name) space. Is it OK or should we optimize it?

AFAIK, it's a tradeoff between the number of msc calls and the space.
As if we try to decrease the number of tables stored in memory, we would increase the number of msc calls and then there would be no point of this JIRA.
Please correct me if I am wrong but this is what I think

My concern was related to the fetch logic where we load all Hive table objects into the memory instead of using the batch iterator.

We also make O(num-tables) calls to load an Iceberg table. Can we optimize here? Then we put those into a separate cache. Maybe iwe could use CachingCatalog instead ?

tableCache.get(tableName, key -> IcebergTableUtil.getTable(conf, table))

I recalled that we would like to retain the try-catch. We intentionally added it to avoid skipping everything when a single expiration fails.
See also: #5786 (comment)